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MEASUI^NG STUdHNTMTITUDES TOWARD 
THE AIR FORCE TRAFFit S^^y COURSE 



I. INTRODUCTION 



The Air Force Tratric Safety Course (AFTSC) C9n*Uted of a 10-hour course of instruction, broken 
down into 10 units, ^covering such topics as environmental ha/.ards, vehicle control, common traffic 
violations, emergency situation?, and mature drjvijri^ behavior. Mandatory for virtually all personnel 
entering the Air Force, this couree wiis being titught at more Uian 140 locations in a standardized, 
multimedia format. Sound/slide presentations and ilims delivered the instruction much like a programiped 
text with student responses called jor at certain pbihU in the program. 

In order to revise the AFTSC as necde(Jy course managers desired a standard method of getting 
feedback on perceived effectiveness from the 'sljud'ent viewpoint. This concern constituted the operational 
problenu Assistance in the development of measurement procedures was requested of training evaluatipn 
speclaUsts in the Air Force rtuman Resources Laboratory , Technical Training Division. ' 

Given certain constraints imposed by the operational setting, it was proposed that a desirable, system 
for monitoring stud||it attitudes/opinions would possess the following characteristics: 

1 . Data sju)uld come from an objectively scored collection method. 

2. Opinions about unit effeotiveness should not be measured for each student on every instructional 
unit, but rather a sampling approach should be taken. 

. 3. A capability for longjtudipal analysis should be available so that long-term trends might be 
detected. 4* 

4. Primary emphasis ^6U1^ be given to determining attitudes tbward (a) overall course effectiveness, 
(b) instructional unit effectivcftais, (c) media effectiveness, and (d) deficiencies in the study environment 
(e.g., lighting and temperature),/ -^'^^^^\ 

5. The capability for comparing responses of subpopulations on tl)e basis of certain 
background/demggraphic characteristics (e.g., male vs. female) should be included. 

6. SUnplicity in tiata collection, tabulation, and analysis should be sought. 

To achieve tixese goals, an approach was recommended consisting of the following steps, (a) 
questionnaire devolopmcnt and construction, (b) operational tryout of administrative procedures, (c) 
validation and reUablllty analysis of the questionnaire, (d) establishment of analysis capability by the user. 
Air Force Directorate of Aerospace Safety, Ground Safety Division (AFSET). 

• This report covers work performed in the first three steps mentioned in the preceding paragraph. An 
implcmental)t* decision will defend on the results to be presented here and the extent to wliich the 
Information Jjiincd from tlie proposed questionnaire satisfies the heeds of course management personnel. 

The- present study focused primarily on establishing ^estionnaire validity. Thus, a discussion is given 
of what the iqucstlonnaire was intended to measure, the characteristics of the sample, the degree to wliich' 
the questionnaire was sensitive to differences in attitudes, reliability of the derived scales, and the Jmiount 
of sucCCwcnCountcred with the mechanics of administration. Secondarily, the actual results of the survey 
arc presented as an example of the kind of evaluation report this critique method can prodilce. 



I 



'I , ■' II. METHOD 



Rationale. In the context discussed here, aj attitude was viewed as a collection of opinion statements ^ 
(Items) with which respondents agreed or disagreed to varying degrees of magnitude. These statements were 
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designed to cover unitary dimensions of attitude (e.g., attitude toward instructional tnedia and attitude 
toward instructor competence). Thus, sU^ong agreement witli items composing an attitude dimension would 
result in a liigii score on that attitude. To the extent thit items were found to relate to oAe another both 
conceptually and empirically (i.e., through observed correlations), an attitude wa$ said to exist. Attitude 
was then measured by summing appropriate Item responses to derive a scale score. In addition to measuring 
attitudes, specific statements about certain aspects of traii^g were elicited. These unrelated statements 
were not conceptualized as attitudes but were treated simply as opinions. , » 

In order to measure attitudes at the course level separately from those at the unit level, two critique 
forms were produced. For the Course Critique, flvtf dimensions of attitude were defined, and items were* 
written to tap the dimensions. The dimepsiohs of interest in the Course Critique were: (a) instructor 
personal relations, (b) instructor technical competence, (c) overall coiWS6 effectiveness, (d) media 
effectiveness, and (e) enviromnental factors. Items which support these dimensions appear in Appendix A. 
Tluee more items, plus^n open-ended section, were added In order to pinpoint specific problem' areas. 

For the Unit Critique, only one dimension, designed to measure instructional unit effectiveness, was 
planned. See Appendix B for a listing of these itertis. A^in, three ad(^K)nariiems vyere Included to cover 
specific instructional features, such as adequacy of ^response time, comjAeteness of coverage, ^nd 
concreteness of examples. 

In addition, eight items of background/demographic information were requested: trairdng base, major 
command, rank, whether the respondent was an officer trainee, sex, status (student or permanent party), 
age, and educational level. 

Validity and Reliability. T4 satisfy the custmnary psychometric requirements, the questiomiaire was 
validated by a demonstration of conlcnt and conitruct validity. The extw^t Id which the .questionnaire . 
items addressed a series S( dimensions logiciQly related to facets of the course is defined as content validity. 
Construct validity 'can be established by citing various types of evidence to check the theory underlying the , 
test. In this instance, the evidence was examined relatinjfto >vhether the measuring instrument was able to 
detect attitude differences When they were expected ^o occur. In addition, factor analysis was used to 
check whether the intended a priori dimensions were actually found to exist when the questionnaire Was 
administered to a representative sample from the student population. As a n^essary precondition for 
validity, reliability, of the questionnaire also had to lie established empirically. In this case, reliability 
referred to the internal consistency of the questionnaire scales. 

These analysis requirements prompted the design and conduct of a validation study. Moreover, such a 
study permitted identification of poor items for subsequent revision or elimination. The pilot study also 
provided a method for testing the mechanics of administration, particularly the unit sampling scheme, 
which was somewhat complex, and for detecting prol)lems students might have !n oomprehendlitg either the 
critique items or the instructions. 

Sampling considerations. Since two major subpopulations were required to take the AFTSC Officer 
trainees and enlisted first-tenji^airmen), it was decided to obtain representation from these two groups. For 
the sake of convenience and because there was no reason to betieve that Lowry airmen were atypical, the 
entire sample con^jisted of first^term airmen at Lowry AFB. Officer trainees at Lackland AFB who took the 
AFTSC as a regular part of their Officer Training School (OTS) curriculum were aly) given the experimental , 
version of the critique and were considered to be representative of the officer subpopulatlon. Although 
USAF ^cademy graduates constituted another subpopulatlQn of officer respondents^ they were not 
sampled, since the primary intent of this study was not to measure the attitudes of any specific subgroup, 
but rather to determine whether the measuring device >Vas appropriate for groups with higher levels of 
education. ♦ , 

Administration, A short pretest was designed and administered for the purposes of det^^^g changes 
in attitude toward the course over time. While this procedure would not be recommended for operational 
use of the qucstk)nnalre, it wasfelt that evidence of attitude change would be useful In establishing validity 
in the present study. The six item pretest was dfi^gned to measure a single dimension -overall course \^ 
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For the enlisted sample, all students v/hi/Took the Al-TSC d.mno f .u 
were given the various oxperin.ental questionnair s ( P els' Uni T"^'^ '^"^ 

LcAvry. the traffic safety course was usually c , pletcJl^^ d L . ^ ?n' "^""^ 
1"^. and 600 responses were obtained to onc'or Zhlr fo^ronre s':;'; 

over a . to .W period. -IVtrwC^ ^^^^ .ve. 

in^ructo: r~ t::rt::::r :rs;':d:nr r^"- ^. ^^-^ - --p"^- 

randondy to one-tenth of the cla Z a ch un T T '""'^ 'J"'' ^"^'^ues 

wen. finished. In this mariner aX ^bs a^^^^^^^^ the students 

Critiques would be available t;/ c^r nit vand i d I k ^'"'^"^^ ^'^^^^ the cc^urse. enougl, Uoii, 
-^ents would not be burdened w^inl-^" ^^^Z^:^ U. Ut ^"^^ ^'^^ ^^^^ 
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eorrrparirrg „„. .1 Llrsirowrt' '"f '^""^ "»« 
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were aclrrally relleererl i„ ,he e p ."^rc , " "' "f "'li'»<l. 
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Table L Lowry Samplo: Background Information 

1 = 



Sex 



Age 



I'ducalion 



• 


(N»300) 


Courts 
Critiqut 
(N'300) 


N 


% • 


N 


% 


Male 




O f 




V 1 


Female ^ • 




1 1 


i / 


t\ 

V 


17 * 


38 


13. . 


3') 




18 


101 


34 


107 


<,36 


19 


C A 


18 




■ 18 


20 


35 


12 


35 


12 


^ 1 


26 


9 


23 


8 


/«: 


16 


5 


14 


S 




10 




8 


3 




lU 




8 


3 




5 




4 


1 




2 


1 


3 


1 


28 and abdve 


3 


r 


2 


I 


Missing' 








1 


Some High School* 


12 


4 


7 


2 


Migli School (iraduate 


10';- 


66 


209 


70 


Some College 


82 


27- 


76 


25' 


(\)llege Graduate 


4 


1 


• 5 


2 


Some Graduate School 


2 


1 


1 




Missing 


1 




2 


1 



Tabk 2, Lacklaii'ti( Sample: Background Information 



Sex 



Age 



l-lducation 



C«t«9ory 



Male 
l-emale 
Missing data 

20 

21 
01 

23 
24 
25 

26 



Pratait 

(N-25) 



College (irailiiate 
Some (iraduate School 
No iJWa 



9 
6 
2 
7 



23 

0 



36 
24 
8 

4 



92 
8 



CourM 
. drHlqu* 
(N«50) 



N 




% 




% 


17 


68 


30 


60 


» 


32 




38 



12 
18 
7 
9 



44 

5 



2 

2 
24 
36 
14 
18 
2 
2 

8l^ 
10 

2 



the atiuHitU br varianco explained, so that only llie siruiiger factors remained. Finally, the Lowry solution 
was rotated to congruence with the hickland solution! 

I ho luilhors wish l(» Ihimk Dr. R(»)(er tVnncll fAl lllU./'IT)/or suHpcstlng thiK.iiiclliud ol unalysis, 
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Prior lo factoring the responses, negatively worded items were reverse' scored to make all items 
amstslent w.th- respect to favorability. The'initial solutions produced nine factors for each student group' 
While there v<,ere son.e differences between these solutions, they were judged to be n.inor. At this point it 
was apparent that three major factors were present: (a) an instructor general lactor. (b) a course 
ellectiveuoss lactor. and (c) a .nedia effectiveness factor (see Tables Dl and 1)2 of Appendix D) These 
three lactc^s accounted lor 8(W, of the response variance in the L.cktand data.4n the l.owry data the 
env.r(mn,cntal factor faiM to emerge as a single entity, but broke .nU as three separate factors (7, 8 
,w.th h.gh loadings on single itenrs. Therefore, it was decided to treat the envjlonmental items as opinion 
.terns, rather than as contributors to a scale, and to drop them Irom future analyses in the investigation of 
<luest,onna,re dnnensionality . Also, l-actors 4. 5. and 6 in the l.,wry data appeared to be fragmented 
versions o l-aclor . the course effectiveness factor. In consideration of these results, it was decided 4o 
constrain the solution in the next step by specifying that only three factors be extracted. 

The three-factor. solutions are presented in Tables 3 and 4 for L)wry and Uckland, respectively It 
can be seen that. Tactor 2 overlaps l-actor 3 to son,e exlent. but items 30 to 33 provided the clearest 
delmilion o l actor 3. namely, attitude toward .nedia effectiveness. Differences still remained between the 
l.owry and l.,ckland solutions, l-or example, in the Lowry data. Factor 2 (course effectiveness ) was strong 
and clearcut. while in the Lackland data, this dimension appeared more weakly as l-actor 3. ' . 

As an objective ..heck on the severity of tftese differences, (he principal-Oomjx^nent Lackland solutiort' 
was rotated to congruence with the Lowry solution. (\)efllcie,Us of congruence were .')7.,8y and 74 for 
•actors 1. 2. and 3 respectively. These can be interpreted as excellent, very good, and moderate 
congruence. » 

Achiitional factor amlyr s. To test the hypothesis that only one fnctorjperceived- course 
eMectivencss) was present in thy pretest, the six pretest items were analyzed for the lowry sample The 
results tended to conilrm the hypothesis. While two factors emerged in an. unconstrained solution, the first 
laj^tttf was predcuiunant and loadecb significantly on five of the six items, accounting for 88% of the 
^wpn^^se variance. Table 1)3 (Appendix D) presents the obtained loadings on the two factors. For further 
imaly^es. it was decided to eliminate the no.Kontrihuting itenf (item 28) from the perceived course 
ellectm-ness scale. Ihe results of a similar analysis with the Lickland sample are not presented In this 
analysis tl,e six-item pretest prmiuced three factors rather than one. and loadings 'did not agree with the 
Lowry Inu ings, It is felt that this discrepancy occurred because the number of students who took the 
pretest in the Lickland sample was quite small (N=25). and the observed result was probably unstable. 

The unit questionnaires were also factor analyzed. Since so few unit questionnaires were completed 

J ''''' '^'^''^ '^"^^y «"«'ysi'' was completed on 

a mtal ol 4M cases. W.th 10 different units represented in the data, it was. decided to remove unit 
dillereiices before lactorinw^otherwise. the factor solution could be inHuencQd to an unknown degree by 
variation within units. Tl/ alternative approach to perform 10 separate analyses and compare them was 
• elected as impractical, and also because it was desired that a more general, overall assess.nent be made of 
the fador structure undedying the unit questionnaire. Tlicrcfore, the data were converted to standard 
scores by a transformation involving the mean and standard deviation of each unit. The results of this 
T M ,^ M ""'^'""'-•"sionality of the unit ciuestionnaire". Factor loadings are presented in 

laWe m. Naming these factors presented some difficulty, however, they appeared to represent^a) attitude 
toward umt objectives, (b) attitude toward unit questionnaires, and (c) attitude toward level of detail. 

In summary, three major factors were found in tl,e Course Critique: (a) an instnictor factor (b) a 
course effectiveness lactor. and (c) a media effectivehes^ factor. Tables 3 and 4 contain the factor loading* 
tor the ( ourse ( ntique. l^liese may be interpreted as correlations of the individual items with the factors 
Some minor differences between the Lowry ancf l^ickland solutKWis were found to exist in the Course 
( rifuiuc'. nie pretest w.is also analyzed, and one gemJral factor perceived course effectiveness was found 
When factoring the Unit Critique, some unexpected factori^ emerged; however, the most important 
lactor overall lesson effectiveness was present as expected. 
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Tahle^, Rotated iHictor Loadingii 
(Lbvry Students, N^300) ' 



t % mm 








\ 


.60 ' 


* 




2 


.71 






1 


.50 * 






4 








5 


".57 






• 6 


.62 






7 


.68 






.8 


.65 






<) 


,73 






10 


' .67 






11 


, .81 






\1 


.52 






\^ 


■W 






14 


■ r 






15 


.83 






1() 


.81 
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17 
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18 


.7') 






1*) 


.48 ■ 
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20 


. .71 
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Table 4. Rotated factor UMdings 
(Uckland Students>i|Pl) 
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Discriminant validity. As .uned by Feclerico (1971). if certain items are able to discrinunale'between 
groups with known attitude differences, tbesi would be more useful items for inclusion in a final version of 
he questionnaire. To examine this p,)ssibility , a discriminant analysis was run on 40 items (rom the Course 
'^'f "^'^ diff^^^J'^'^te the officer aud enlisted groups quite well 

cem«>K*^bc.ng 2.63 au<l .64, respectively^ In addition, 94% of the ca'ses were correctly predicted as to 
group nu^mbersrfip by tin, discriminant fimction. Thirty^cven of the 40 itc.ns had significant discriminatiug. 
.p<.wer. These arc listed in Table 5 along with their standardized coefficients on the discriminant, function 
Tliesc coelficents roprq^the relative contribution ofi^^ach "item to the discriminant function. Using an 
Jbsolute value of J5 .^.arbitrary ^t„ff, 18 of the 37 items made relatively strong c^ontribution.. 
nearly, these would be candidates for retention if a shorter fonn of the questi(,nnaire were dejired. 

Other cviikmv of vatidity. |f tlie questio.mair* was sensitive to differences in attitude which arc 
known to exi.st. then one c(,uld argue that the questionnaire was validly measuring what it was designed to 
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iMcasurc. It was widely Icll by course numagcrs and instructors tliiit the officer group would disltke thfe 
rq^titive aspects of tlie course and, in fact, Would perceive tlie entke course lesjHavorably than the enlisted 
.^roup. IrvJuldition, if the (piestionnaire was sensitive to changes in attitude ov>cr time, this would be furt'W 
evidence of c^istf^t validity. To clicck these hypotheses, a scale was.constt,ucted by suninung the five 
Itcm!^^^ which hud been identified as the primary factor in the pretest. This scale (items 21 . 22, 23, 25, and 
26 from the Course Criti(|ue) can be lluniglit of as* measuring perceived course effectiveness. Using 
perceived course effectiveness as the criterion measure, a two'-way factorial aiudysis of varinnce was 
performed with test time as one factor aiid of ficer or'tjnlisted group as the other. T>^ty-i#ne subjects were 
dropped fr(Mu the analysis because scale scores could not be calculated due tiPkne or more missing 
responses. Results^are depicted graphicallv iiu-l'igure 1. The significant interaction showed that Jthc olfllcer* 
group v/i\H less favtuable toward the course before taking it and became even more unfavorable after h^iving 
taken it. Table 6 contains the ANOV A sunnuajy t^iblc. ^ ' 4 ' v 
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h'iKurc I. Perceived course effectiveness: Differences between 
officers and enlisted personnel. 



Tahlc 6. An;}Iysis of Vnrtanee Sutntnary 
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Rdiafnlity. AllliMUgli factor atuUytic procedures were helpful iti idctitifyini^ itctus for iticlusioti on 
scales and ni naniit.g these scales, it was still tieeessary to check the reliability of the (|uestionnaire' 
yuestionnairQ rehability could not he cahculated using all items and determining ititerftal consistency since 
the mstruM.cnts wore not iutchdcd ft) be. nor did tlvcy prove to be, n.casurit.g only one dimct.sion A more 
appropriate n.ethod was to co.uputc the /cliability of e*.h scale using Cronbach's Alpha to estimate 
• in^crt.al consistency reliability. Table 7 prcset.ls these tesnlts. Low reliabilities were observed for scales 
within tin; Unit Criticiue. *«.'.' 

^ Survey results. It. Tabic 8, means, statuiard doviaUpn's, atui values arc given for items of the'Course 
( r»M.o, broken down by ofRcer vs. enlisted. Those ito.u.s ««)did not fit \n the three scales are prescnftd 
•xPn-blc ,9. .with percentages of resp^^^dents in Ihb' varlou/espi^se categories. In these instances, chi-suiiarc 
tests were Viuule to determine whether the distributior/of responses dificred signilTcantly bctwoc.. the two 
groups. Duo to expected cell frccfuencicS less than 5. ^Im^gtuics In iton.s 41 to 4.3 had to be collapsed to 
nutt tiK; statisti^jal requirctucntspf the chi-S(|uare test. A nutuber of differences were observed Scale scored 
are reported in Table 10. 
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Tahli^, Internal ( onslstency Reliability estimates 
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Table Item Difference^: Couwe Critique (Items 41 (o 43) 
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Table 10. Scale Differences: Course Criti(|ue 
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Item and Scale scores on the Unit Critiques are given in Table 1 1 for the total satnplc broken down 
by unit. Tliougji differences among units were noted for items 8 and 10, no differences were found iwrcxss 
the varioui unil^ for the unit effectiveness scale. 

Tho opcn-«ndcd portion of the questionnaire proiluccd a number of conipients/rec<)tnmcndatioii.s 
which were grouped iato categoric* and listed In Table 12. A much larger percetUagc of tt\<)fncct group 
chose to comment and their comments were predominantly negative. ' 



Tabh' 1 1. Item and Scale Scores: Unit Critique 
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Table 12. Content Analysis of Recommendations 





Comment 




Officers (N=50) 


/ — ' 

Very limited intellectual 
apjKjal; dry; boring 

Should be eliminated because 


33 




It takesvup valuable time 


30 




I oo long 


10 




No comment 


\ 


Enlisted (N=300) 


Acquired useful information 






or enjoyed class 


1 ) 




Boring 


pi 




Too long 


1 ^ 


i 


RcDetltioiis 


/ 1 <J 

/ 


\ 


liliniinate or sliorten course 
because most people liave 
iuivi.li uiivLT 5 cuucaiioti 


1 ' ' 




Pace too fast to mi^wer 






questions / ) 


O 




IVoblenis witli media coniomiMit / 






slides out of order, sound ' 




1 


, inadequate 


21 


t 


Instructor sliould have greater 


• 




role in course 






Too many hours pe^day 


1 1 




Update films ^ 


() 




More shocking and dramatic 






visual presentations of 




y 


accidents ' 


- 10 




No comment - 





( 



IV. DISC USSION * ^ * 

^ • 
Altlun.Bh the empirically derived factors diil not exactly match the a priori intended factors a usable 
s<t of Uimenjfions was found to bd |)resent in both the Course and Unit Critiques. Wl.ile previous' reiearch 
(Katitor. V.tola. & (mi.m. 1977) with the tw separate Instructor scales (instruptor technical competence 
and ini^tructor personal relations) indi .cd that measures could be obtained on both these dimensions 
in retrospect, the fact that only one factor emerged from the analysis was understandable. Tliis result can 
probably be cxplainyd hy circumstances peculiaf to the Ai-TSC. The relative brevity of the course coupled 
•with the- fact that instructors were ^ot Ihe principal medium for transmitting inforn^ation, probably 
hJUnpcrcd the respondents' ability to distinguish two separate facets of the instructor \oIc. Given the 
relatively minor role oP the in.structor in provitfthg instructional content in the multimedia AFTSC, little 
has been lo^t by restricting measurement to One general instructor factor. ' 

' With regard to the course effocUvencss factor, tnost of the itettft which had been written to measure 
this dimension were successful. Items 8 and ^) wqrc notable exceptions and could probably be dropped with 
little loss in information. 



Uss successf ul were itents ileslgned Ip measure "the media effectiveness dimension. Only four highly 
stable and interrelated items coiUrlbuted to measujing this dimension. Such a small number of items 
approached the lower limit for constituting a scale. -^..^l^ ' 

In the Unit C'ritiques, only one factor had been anticipate^ unit (or lesson) overall effectivenesS-bUt • 
three were found upon examining the data. However, the intended factor was the strongest and the others 
may likely be dropped withoul^defcating the purpose of the Unit Critique. 

Dissimilarities found in the Lowry and liickland factor solutions, while large enough to cause some 
concern, were mM^vicwed as s<jrious enough to warrant separate scale construction or revision to the 
dimensions* as presently conceived. Sampling error and the fact that the Lackland sample was relatively 
small may have .accounted for the obtained clifferenccs. Moreover, at least moderate congruence of t^ie 
factor solutions was achieved in the present siimples. 

From a psychometric point of view, it is^sky, if not improper, to use a single Item fOr assigning a 
numerical score which is intirpreled as an alt1^||t measurement,, The reliability of a one-item measure is 
indeterminate uSing inlcrnal.'gonsistcncy estirfTation procedures and would be theorelic^ly low' In any case. 
Thus, it is reconuuended that attitudes toward instructor, course, and merfia effectiveness be measured by 
ineans of multiple item scales. Those developed in this study were shown to have adeq^uate reliability and 
good construct validity. Unit effectiveness also appeared to be a reliable and valid scale. Thera. is, however^a* 
defensible way of interpreting single item responses. One.can obtain a great deal of information by vie^vjng 
siJiglc ilcim as opinion statements and comparing frequency distributions {9 identify group differences in 
response patterns. If an unfavorable response category is picked more frequently than could be accounted 
for by chance among a sufficiently large number of peoj)le, this can be taken as evidence that an 
unfavorable .situation exists. Chi-square tests can be used to determine the stalistibal significance of deviant 
response patterns. . • m, - - , 

V. C ONCLUSIONS AND RECOMMENDATION 

Conclusions * • « * , 

t 

1 . Tlie experimental versions of tlie Course and Unit Critiques demonstrated adequate content and 
construct >ralidily. 

2. Reliiibilily was found to be excellent for all derived scales of the Course Critique and satisfactory, 
tor the most imix^rlanl. scale of the Unit Critique. " 1 

3. Scales were judged to be approf)riate Tor measuring attitudes of two .major subgroups who took 
the Al*TS(\ ; ' 

^ 4. Fnvironmculal problems can be identified by specifia opinion items.. - 

5. Administrative procedures for collecting Unit Critique information were marginally successful. 

6. No differences between units with regard to effectiveness were observed in the present sample. 
That is, in genefal, personnel participating in the survey felt that alHhe units were fairly effective. . 

7. As a whole, the AFTSC was pirceived favorably by the airman ^oup an dnather unfavorably by 
the officer ^roup with attitudes in thti latte^ group getting more unfavorable ^fter exposure to the course. 

8/ While these findings, strictly speaking, do not generalize outside of the Content aita^of Air Force 
Traffic Safety, mmiy of the questions appear to be appropriate for other subject ihatter with suitable 
niodifications. When such applkati^s are cohsidered, checks on the reliability and validity of the 
questionnaire may be advisable.^ " , . " ^ ** , 

Recommendations ^ ,^ ^ 

1. The tJnit Critique form should^c adopted as presently constructed. ^ 



2. The Course Critique form should be adopted after modifications to shorten the Instructor scale 
The revised inrtructor scale should contain the foUowlng ten items: 2,4, 5, 10, 1 1, 13, 14, 15, 16, 17. * 

3. In future versions, consideration should be given to lengthenimg the media effectiveness scale by 
adding similar items. ' ^ , ' 

4; Written instructions for adminlstraUon of the Unit Critique should include a comprehensive 
examj^le to clarify the procedure. ' . ^ 

5. - A procedure for analyzing daU should be developed. Analysis capabilities should iiipludc as a 
minimum: (a) calculation of descriptive 9tatisti^ on any possible breakdown of the population, (b) the 
labiUty to produce frequency, distributions the various item response categories, (c) the ability to 
BMregate data from various administrations of the survey, (d) two-\»jay or higher analysis of variance, (e) 
crossrtabulations with various statistical measures of association produced as part of the analysis. 

6. Personnel who would analyzS survey results should be identified and trained", if necessary to 
perform desired analyses. - ' 

■ 

7. hi operational use, norms should be calculated and trends monitored. 



REFERENCES , 

Federico. P. Identifying item ^validity indices utilizing a multivariate model. AFHRL-TR-71-16, AD-729 
763. Lowiy AFB, CO: Technical Training Division, Air Force Human Resources Uborato'ry, April 
1971. >^ 

Kantor, J.E.. VitoU, B.M., & Guinn, ^.^Development and validation of the Air Force Technical Trdming 
Student Survey. AFHRL.TR-77-27(0, AD-A042 967. Brooks AFB,TX; Hrsonnel Research Division, 
Air Force Human Resources Laboratory, June 1977. 
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APPENDIX A: QUESTIONS COMPRISING THE COURSE CRITIQUE 



} 



L Ineffective 


Effective 


2. Knowledgeable 

3. B6ring ^ 


\ Ignorant 
\ Interesting 


4. Dependable 

5. Disorganked- 


Undependable 
Organized 


6. Unsure 

7. Convincing 


^Confident 
Unconvincing 


8. Unprepared 


Prepared 


9. rntelligem 


Stupid 


10. Inefficient 


Efficient 



11. Encourages • Discourages 

12. Criticizes ^ Praiies ' 
'3- Fair Unfair 

14. Impatient Patient 

15. Considerate Inconsiderate 

16. Hinders \ Helps 

17. Friendly^ Unfriendly 

18. Supportive ViostUe 

19. Ridicule^ Compliments 

20. Cooperative Uncooperative 
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The following items pertain to the program taken as a whole. Circle the response which most nearly 
fits your opinion. 



SD » 
D - 
N • 
A » 

SA - 



Strongly Disagree 
Disagree 

Neutral, or No Qpiniqn 
Agree- 
Strongly Agree 



1. My approach to driving will probably be 

affected by having taken this course* 

2. I enjoyed the course/program. 

3 . Taking this course was a waste of time. 

4. The course objectives were appropriate. 

5. I learned a number of things I did not know. 

6. I have a better understanding of traffic safety^ 

(rules, precautions, hazards) sihce taking this 
course. " V 

7. I tKInk others of my genend background would 

profit from taking^this coarse. 

8. This course addresses a real Air Force problem. 

9. The program made me feel more confident about 
my driving skills. 

I liked the automated type of presentation used 
in this course. 
The learning strategy used in this course (rule- 
example-practice) is a good way to teach. 
The slide presentations used in this cotirse were 
" effective in getting across the material 
13', The films used in this course vftxt effective in 
getting across the material. 

14. I found the fllnvs entertaining and enjoyable. 

15. I found the slide presentations entertaining 

and enjoyable. 

16. The way the instructional content was presented 
was apprppriate for this kind of subject matter* 

Gassroom ventilation was about right. 
The classroom was not dark eh&ugh for audio- 
visual presentation. 
Seating ivrrangement allowed me an unobstructed 
view of the screen. 

Noise level did not distract me from the program. 



10 
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17 
18 

19 

20 



21 . Most of the.nbise came from: 

a. outside the classroom 
^ b. htside the classroor 



c. no noise problem 
22. Classroom lemperature was: 

a. hot 

b. warm 

c. just rigtit 

d. cool 

e. cold 





N. 


A 


SA 


2 


3 


4 


5 


2 


3 


4 


5 


2 


3 


4 


5 


2 


3 


4 


5 


2 


3 


4 


5 


* 








2 


3 


4 


$ 


2 


3 


4 


5 


2 


3 


4 


5 


2 


3 


4 


5 


2 


3 


4 




z 


•3 
J 


A 

4 


< 


2 


3 


4 


5 ^ 


2- 


3 


4 




? 


3 


4 


5. 


2 


3 


4 


5 


2 . 


3 


4-- 


5. 


2 


3 


4 


5 


2 


3 


4 


5 


2 


3 


4 


5 ' 


2 


lit. 3 


4 


5 



1^ 
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23. This course was given at a bad time of the day (e.g. just after lunch, etc.). 
• a. strongly agree 

b. ' agree 

c. no opinion 

d. disagree 

; e. strongly disagree 

Ifyou agree, tell why: ^ 
SECTION V Recommendations for Change 
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APPENDlXJ^Ji^^ CpMPRlSlNC^HE UlsJIT CRITIQUE 



In this section please answer the questions only with rjspcct to one of the units within 
Unit you were assigned to rate: 



SD = Strongly Disagree 

D = DlsaRree 

N « Neutral, or No Opinion 

A = Agree 
SA Strongly Agree 



t. 



t. 



SD D 
1 2 



1 . The questions were so simple that they, were 

an insult to my intellegence . - 

2. The objectives of this unit were clearly 

. stated. 12 

3. Most of the time I understood what was being 

asked by the question slides. I 2 

4. In gelneral; the Inltritiation presented in this 

uftit was up-to-date. 

5. I feel that thi? unif clearly achieved its 

oT)jectivesv / ' ' 

6. This'unit was full of needless detail. 12 

7. This unit contained too many technical terms. ^ * .12 

8. The amount of time provided for responding to the slides was: 

a. far too short V 

b. too short 

^ c. just about right 

d. too long 

e. far too loii^g 

9. The amount of coverage ip. this unit .was: 

a. far too little 

b. too little h- 

c. just about right 

d. too much 

e. .far too hiuch 

10. The number of concrete examples given in tliis unit was: 

a. far too few 

b. too few s 
jiist about right 
too many 

0. far too many ^ 



APPENDIX C: QUESTIONS COMPRISING THE PRETEST 



The following items pertain to the Traffic Safety Course you are sched^led 
resj^nse which most nearly expresses yuu; opinion. 

SD ^ Strongly Disagree 
D Disagree 

N = Neutral^ or No Opinion ' ^ 

A ^ Agree , \ * 

SA =^ Strongly Agree • 



to attend. 



Circle the 



SD D 



1. My approach to driving will probably be affected 

as a result of this course. 

2. I am looking forward to takin^this course. 

3. I think this course will be a waste of time. 

4. I will learn a number of things fiom tills course. 

5. I will have a beHer understanding of traffic 

safety (rules, precautions, hazards) upon com- 

f)letion of the course. 
t> 

6. course will address a real- Air FQrce problem. 



2 
2 
2 
2 



2- 
2 



N. 

3 
3 
3 
3 

3 . 
3 



A . SA 



4 
4 
4 
4 

4 

'4 



5 
5 
5 
5 

•-5 
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APPENDIX D. ADDITION'AL FACTOR ANALYSES 



Table DI. Nine Factor Solution of Coume Critique (Lowry) ^ 



lt«mi ' Factor 1 



Factor I 


Faotor 3 


Factor 4 


• Factor ft 


Factor % 


Factor 7 


^Factor 1 


Factor 9 


.155 


\022 


.095 


.058 


> 

.691 


' - .113 


.092 


.1 19 


' .067 


~ .039 


.032 * 


- 041 


.275 


OKO 


002 


064 


315 


.055 


336 


^ .049 


154 


yfi94 




- 034 


701 








f Oft 


{ OIR 


01 1 


0^4 








.J 


077 


i- own* 


ORS 

— .UO J 


040 


•U JO 


* — .U 1 J 


.t6 1 O 


41 Q 

.HIV 


1 ^fk 
.1 JO 


07R 




0^ 1 ' 


.255 


.047. 


-..284 


' -.007 


• .062 


.061 


-.168 


-.132 


.015 


-.053 


-.168 


.412 


.010 


-.047 


.Q75 


-.055 


.169, 


^ .062, 


-.205 


.001 • 


.039 . 


.259 


.007 


-.101 


.193 


- .075 


' .129 


,.155 . 


.439. ' 


.055 


-.074 


•^•.081 


.142 


-.020 


,r.054 


.004 


.132 


.038 


-.019 


.076 


, ,.177 


-.050 


.187 


.092 


„ .01^ 


-.080 


.H3 


„083 


■ .118 


.031 


-.006 


-.OOS 


.023 


~^.in 


■'.113 


.046 


.024 


.079 


.090 * 


.174 


.101 


-.0?3 . 


.065 


-.063 


.109 


.050 


.086 


-.009 


-,080 


- .032 


.056 


.074 


.169 


• .007 


.043 


. .082 


.074 


.064/ 


.038 • 


.013 


, .940 


.151 


^.031 


-.043 


-.097 


-.01 


^.010 


.021 


.079 


.005 


.131 


.104 


--.101 


.020 


.036 


.052 


.161 


.008 


.291 


.■222 


.114 


.150 


-.143 


.007 


-.001 


.214 


.026 


-.089 


-.016 


-.039 -> 


.036 


.014 



Initroctor 

1 

2 . 
3 
4 
S 

. 6" 

7. \ 
8 
9 
,10 

^12 

13 

14 

15 
. 16' 

17 

19 
20 
Courae 



,530 
.688 
.48-1 
.676 
.516 
i56 
.692 
.613 
.745. 
.614 
.805 
.512 
.811 
.781 
.860 
.795 
.829 
.796 
.446 
.745 



1 . 


. ;i29 


.629 - 


.074 • 


.031 


.047 


:1070, 


.095 


-.007 


':049 


• 2' 


.238 
• ill 


.644 


.169 


. .122 


,-'-.108 


.043' 


-.076 


".108 


-.058 


3 


.680 


.069 


.005 - 


' -.024 


-^.065 


-.024 


.172 


.048 


•4 


.055* 


.513 , 


.1 18' 


■ .in*- 


-.033 


.118 


.096 


' -.lf6 


.005 


/ 5 ' 


.090 


.754 


.010 


.050 


.021 


.083 


.084 


♦ r-.064 


.025 


6. 


.209 


.742 


'.069 


-.102- 


.000 


-.025 


.019 


.005 


.085 


7 


- . .057 


.650 


.213 


.059 


..107 ^ 


-.040 


.013 


-.086 


.098 


3 • 


.01 3 / 
.039 


.265 


.181 - 


-.048 


-.027 ■ . 


.005 


.005 


.089 


' .165 


9 


.379 


.222 


.1^4 


-.035 


.101 


.220 


.083 


.003 




.160 . 


^21 . 


.683 


-.097 


.039 ^ 


.027 


/-.092 


.101 


.004 


11 ' 


.017 


,426 


.6^ ' 
.714^- 


.052 


.078 


-.072. 
.063 


.11*. 


.054 


.073 


12 


.018 


.351 * 


.160 


.013 


.121 


-.258 


.105 


13 


.051 


.449 


.473 ' 


.287 


-.053 


-.041 


.129 


.043 


. ^ 


14 


' .1^8 


.604 


.280* • 


.28$ 


-.018 


-.039 


' .022 


• .299 


-.^372 


is- 


.100 


.533 


452 ■ , 


.204 


-.010^ 
.069*^ • 


.016 


' .004 


.240 


-.354 


le ' 


.041 


.475 


■.:224 


-.025 - 


''035 


.033 


.153 


.057 


17 


.047. 


.131 


.027 


-.005 


-.019 


.024 


.M8 


.513 


.120 


18 


.044 


.080 


.095 


.030^; 


-.006 . 


'.018 


.026 


.096 


.470 


19 


.031 


' .065 


.051- ^ 


.049 


- .000 


-.036 


• '.614 


> .090 


.002 


20 


.J)86 


M41 ^' 


.299 


-;167 


-.082 


.086 


.323 


.093 


, .198 



V 
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Table D2. 



Nine Factor Solution of Coune Critique (UckUnd) 



J) 



lUini Factor I l^ictor 2 



-^7 

8 
♦) 
10 
■ II 
12 
13 
14 
1*5 
16 
1.7 
1^ 
19 
20 

Course 

I 

•» 

3 * 
. 4 

5 

6 

7 

8 

9 
10 
11 
12' 
13 
14 
15 
16 
17 
18 ■ 
19 
20 




VMot * : s^iotor » Victor » Fiotor 7 F»otor 1 ' • r^tor ■ 



/\ 
/' '''\ 




• .079 


.579 


.018 


.023 


.038 


.290 


.UV.1 \ 


.00^) 


.295 


.386 


.027 


- .Oltl 


.017 


-.082 


.OS 3 I 


.057 


.017 


.540 , 


.136 


.044 


.108 


.044 


-.0/1 \ 


.ool 


' .359 


.462 


.2b3 


.193 


..201 


-.21 1 


.072 f 


.OLI 


M58 


.026 


.128 


.362 ^ 


.081 


• .131 


Af rZ 


.039 


:040 


.125 


.036 


.190 


.126 ' 


- .013 


.1 30 


•;05 1 


,».168 


.004 


.1 17 


.131 


.208 




.100 


.089 


.038 


.021 . 


.299 


.092 


;t)58 


.U 1 U 




.059 


.J 49 


.037 


.128 


- .064 


.044 


.1 53' 


\rf\ 




Inn 
.JUL) 


.155 


* .358 


.132 


.065 


X)70 


.210 


.013 


.026 


.035 


.1 65 


.083 


.328 


.121 


.057 


^266 
^90 


' .036 ♦ 


.199 


.080 


.068 


.476 


.110 


.021 


.0<i5 


.002 


.103- 


-.029 


-.309 


.040 


.088 


.004 


!073 ' 


.038' 


.078 


-.053 


.249 


.014 


.071 


■ .050 


.060 


•.ooj 


.097 . 


.035 


- .040 


.010 


.018 


.027 


.12S 


.076 


.051 


^.01 1 


- .057 


.172 


.015 


.059 


^*.035 


.089 


.126 • 


W3! 


-.145 


.048 


'.088 V 


.086 


V.068 


.142 


.165 


-.016 


-.067 


.008 


.081 


-.010 


-.053 


-.046 


.203 


.279 


.394 


.065 


.055 


.044- 


.115 


.072 


-.046 ' 


.022 


.010 



.170 . 


.113 ■ 


.665 


.Q53 


.113 


.15 If 


.315 


.100 


.156 


.184 


.239 


.312, 


.057 


* .267 


.127 ' 


.063 


^540 ^ 


.169 


.030 


.1 18 


.462 


.127 


.262 


-.038 


' ^-.044 


•II68' 


.182 


.059 


.469 Ir 


. . .269 


*«.054 


.043 


.134 


'.091 


.150 


.301 


.060. 


.198 


.741 


.30^5 


.150 


. .18^ 


.146 


.039 


-.326 


.013 


:i 26 


.825 


.150 


.064 , 


.029 


.010 


.103 


-.135 


.053 


.,,056 


.781 ' 


.* .291 


.042 ' • 


.181 


.035 


.055 


.1 18 


.30^) 


.072 


.06i 


^/028. 


1012 


• .123 . 


.637 , 


-.027 


-.044 


.011 ) 
.008 ' 


.017 


.10*) 


.101 


i230 • 


.037 


.177 


■„ .184 


.071 


.607 


.178 


.125 


/b31 


.348 . 


.175 


.221 


.042 


.181 * 


.781 


.105 


.123 


.Il3i 


.198 


-.1.35 


.183 


.042 


• .023 


'.857 


.1 18 


.159 


.067 


.024 


.080 


.241 


.086 


.062 


.907 


.044 


.life 


-.141 


-.064 


'.208 ' 


.121 


.004 


.001 , 


.382 


• 1 36 


.066 


.163 


• .086 


.001 


» .825 


.048 


.022 


.363 


.105 


.00<) 


.035 


- .024 


.013 


.90| 
.392 


.044 


>.064 


.457 


.286 


.385 


.019 


.054 • 


.229 


-.137 


.091 


.094 


.089 


^81 ■ 


.024 


.168 


-^02 


.003 


.019, 


.414 
' .11} , 


.006 


.150 


1644 


.068 


.021 


..274 


-.005 


-.097 


.079 . , 


.327 


.205 


.455 


.149 


.599 


.035 


• -.025 


.110 


.149' 


.0.58-' 


.101 


.100 


.826 


.123 


•-.040 


.003 
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Table D3. Factor Annlysis 
of Pretest (Lo wry) 



lUmt Factor 1 Factor 2 

^ 

1 .611 "'207 

2 .765 .050 

3 .681 .066 

4 .630 .522 

5 .560 .642 

6 .003 .291 . 
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Table M. Factor Analysis of Unit 
Qu^tionnaires (Combined Group) 





Factor 1 


Factor 2 




• 1 


.082 


.575 m 


r.ooo 


2 


.574 


.044 


.129 


3 


.471 


.411 


-.227 


4 


.618 


.147 


.085 


5 


.664 


.233 


.051 


6 


.183 


.654 


.409 


7 


.111 


.055 


.580 


.... « ^ 
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